Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: AlecWest Date: Jul 27, 2006 11:22am
Forum: feature_films Subject: ground rules for uploading film

Hello. Note - I have a broadband cable connection with 7mbps download and 3mbps upload ... and I have a few questions. I went through the process to upload a public domain feature film to the archive in four formats (smaller files uploaded before bigger ones):

1) RM 12 mb)
2) WMV (141 mb)
3) MPG (832 mb)
4) MPEG2 (4.35 gb)

According to the guidelines, I was given 48 hours to upload these files. It wasn't until I got to the MPG file that I noticed 3 things:

1) Even though I set SmartFTP to upload files using three upload threads, archive servers limited me to a one-thread upload.

2) Even though I set SmartFTP speeds to "unlimited," archive servers throttled my upload speed to an average of 44.5K/s.

3) Even though I set SmartFTP's "file exist" settings to "resume," archive servers "overwrote" uploaded files.

Despite these limitations, I decided to dedicate my computer to the task. I got the first 3 files uploaded and was uploading the 4.35 gb MPEG2 file. But shortly after uploading over a 24 hour period, the archive's server disconnected me ... meaning that all that time I spent uploading it was wasted.

I called Archive.org on the phone to try to see if I could get my deadline extended beyond 48 hours. And all I got was a "just post your problem to the forum" response.

Well ... here I am ... in the forum.

Questions:

1) Can the 48 hour deadline be extended to 96 hours ... or at the very least 72 hours?

2) Can servers be changed to accept more than a one-thread upload?

3) Can the speed at which servers throttle an upload be increased?

4) Can servers be set to accept "file resume" on uploads in case of disconnections?

FWIW, I was uploading the 1932 film, "If I Had A Million" and was assigned this FTP server to upload the files:

ia301236.us.archive.org

...and into this directory:

/If_I_Had_A_Million/

Because I could get no help on the phone - and because your site says files existing in that directory would be logged after the deadline passed - and because there was no way I could re-upload that MPEG2 file before the deadline passed, I've deleted all the files in that directory - except the 2 that were created automatically.

Let me know how to proceed.


This post was modified by AlecWest on 2006-07-27 18:21:43

This post was modified by AlecWest on 2006-07-27 18:22:56

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 28, 2006 2:57pm
Forum: feature_films Subject: just a PS on bandwidth "throttling"

As mentioned earlier, I upload via FTP to many servers ... even overseas. And even a German server I upload to lets me upload at 178K/s. I called up Comcast to see if they were throttling my connection to the Internet Archive back to 45K/s. They said no.

Of course, they could be lying. It's possible that certain remote servers known for high-bandwidth uploads and downloads are on Comcast's "s*** list" and that they routinely throttle connections to those servers. But, I've never encountered slow speeds like this before ... except when connected to one of those sites that reroute DNS info to allow people to use their home computers like a server.

BTW, the average is now down to 42.22K/s.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Jul 28, 2006 3:56pm
Forum: feature_films Subject: Re: just a PS on bandwidth 'throttling'

Perhaps you're running up against something like this then, but on the upload side?
http://www.archive.org/iathreads/post-view.php?id=62445

(Just guessing, as I'm not an IA engineer.)

This post was modified by Diana Hamilton on 2006-07-28 22:56:16

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 29, 2006 6:16pm
Forum: feature_films Subject: Success (in part)

I've uploaded the 3 media files (MPEG1, WMV, RM) during the "check-in" process and will attempt to upload the MPEG2 file so the upload completes next Tuesday morning, August 2nd.

However, I noticed 2 problems. The first one might be fixed by the time you read this. Then again, maybe it won't. I copy/pasted my film description info from a word-processor file. Well, it didn't like some of the text formatting and the description information looks "weird." However, I noticed the new "Edit" feature - and replaced the word-processor text with plain ASCII text. I got a message on the next screen telling me my edited text will take "several minutes" to resolve on the details page. Of course "several minutes" is a relative term - and as of now, the correction hasn't been logged or completed.

But, there's another odd error. If you click on a link to the WMV file from the details page, it takes you to a blank page. And if you click on the WMV file from the "All Files - HTTP" page, it takes you nowhere. But, the FTP page works. There must be a reason for this but I can't figure out what it is.

Here's the details page:

http://www.archive.org/details/If_I_Had_A_Million_1932

Reply to this post
Reply [edit]

Poster: k-otic Date: Jul 30, 2006 1:11am
Forum: feature_films Subject: Re: Success (in part)

At the moment the only files which are available for me are the MP4 streams derivations from your MPG1 file

Reply to this post
Reply [edit]

Poster: k-otic Date: Jul 30, 2006 2:13am
Forum: feature_films Subject: Re: Success (in part)

forget about my last post the problem was on my side ... disk full... ;)

Reply to this post
Reply [edit]

Poster: LordOfTheExacto Date: Jul 28, 2006 8:12am
Forum: feature_films Subject: Re: ground rules for uploading film

I don't have any practical help to offer, but after looking that film up on IMDB, I sure hope you can get it through. It looks hilarious.

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 28, 2006 9:55am
Forum: feature_films Subject: Re: ground rules for uploading film

Thanks. Sometimes I wonder if Archive.org doesn't go out of its way to prevent people from uploading. Out of all the features I mentioned in my first post, the ability to "resume" a file upload if interrupted would be the logical one to have it place ... yet it doesn't seem to be possible. There are probably a lot of people who'd like to contribute feature films to the archive ... but there aren't a lot of people willing to "dedicate" their personal computer to a task that would take 28 hours 45 minutes at the current throttled upload speed. And of those that would be willing to do so (like I was), it's disheartening to cross the 24 hour mark only to have the server disconnect ... especially when the ability to "resume" doesn't exist.

Hehehe, perhaps there's some rarified "cabal" of people out there who get premium upload access to archive servers. All I know is that I'm not in the cabal.

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 1, 2006 6:32pm
Forum: feature_films Subject: Re: ground rules for uploading film

This will be my last post here for a while.

The 24-hour cutoff I experienced is not related to SmartFTP. Another user in the forum has experienced the same thing - and he/she uses WS_FTP software. So, the cutoff is server-related.

My ISP is not slow ... and they are not throttling my speed back to 42-45K/s. I called them up to ask. They said no. I routinely upload to domestic servers at 10 times that speed ... and even one German server allows me 178K/s.

Usually, when people are willing to donate something to a worthy cause, the entities accepting a donation will go out of their way to assist in the donation process. I swear to God that this is the first place I've encountered that puts roadblocks in the way of a donated film ... throttled speeds and pre-designated server cutoffs.

FWIW, I did not give up on this process right away. I finally got a response from an Archive support person who suggested the reason I couldn't edit the screwy text (my fault it was screwy) and upload the MPEG2 file into the original directory was because I hadn't "checked out" (I most certainly did). In any event, he checked me back in and I attempted to re-upload the MPEG2 file - later, to edit the screwy text before checking out. However, on this new attempt, my upload speed was throttled back to 22K/s - making it absolutely impossible to upload the 4.3gb file within any 48-hour window - and this assumes the 24-hour cutoff didn't raise its ugly head in the middle (a BIG assumption).

I have asked Archive support to delete the entire film directory for this file. And at some point in the future, when and if the Archive removes these roadblocks to donation, maybe I'll return and upload the film again - and other films as well. But as long as these roadblocks exist, it's frankly not worth my time and my computer's time to fight with a stone wall.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Jul 28, 2006 11:17am
Forum: feature_films Subject: Re: ground rules for uploading film

Even though I set SmartFTP speeds to "unlimited," archive servers throttled my upload speed to an average of 44.5K/s.

This is the first I've heard of any 45K/s upload cap on the archive.org side (although that *is* a common cap size for a typical home cable broadband plan, like say comcast). Did IA folks tell you on the phone that that was a limit, or how did you determine that?

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 28, 2006 12:45pm
Forum: feature_films Subject: Re: ground rules for uploading film

Diana wrote, "This is the first I've heard of any 45K/s upload cap on the archive.org side (although that *is* a common cap size for a typical home cable broadband plan, like say comcast)."

SmartFTP displays that average. FWIW, I created a new directory and am making the attempt again. And yes, same average. But, I've never encountered that upload speed at any other site I upload - and I upload to a lot of sites via FTP.

In any case, I did notice one improvement. The server I'm currently uploading to (ia301316) accepts the "resume" if disconnect occurs ... but the initial server (ia301236) did not. And, swear to God, I didn't muck with my SmartFTP settings between servers. The "resume" is just working when it didn't before.

Current upload speed average is 44.17K/s with 27 hours and 26 minutes to go ... and that's just on the MPEG2 file. After it completes (if it completes), I've still got an MPEG1 file, a WMV file, and an RM file to upload into the same directory ... before I visit the "check in" page.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Jul 28, 2006 12:57pm
Forum: feature_films Subject: Re: ground rules for uploading film

Well, if the whole fileset can't complete in 48 hr, but any given file *can*, you could do it this way: Successfully upload a file in a 48 hr period, and check it in. Then, go to edit details> item manager> checkout> upload another file as per instructions that time> checkin on that one. The repeat checkout> up > checking with next file. (6 files, 6 48 hr windows if you want.)

If a given file is so large that it can't complete in 48 hr by itself, breaking it into parts somehow might be better? That might be better anyway, because on the d/l side, others might have problems successfully completing *really* huge single downloads themselves.

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 28, 2006 1:06pm
Forum: feature_films Subject: Re: ground rules for uploading film

Thanks. I'm going to leave it running through the night (I work graveyard shift) and I'm guessing the upload should complete around 4:15 PM PDT ... unless I get disconnected prior to that time. If I do, and if the resume feature continues to work on the new server, I should be able to do everything within the 48-hour window.

Still (ahem), it would be nice if the archive gave people, say, a 72-hour window instead of a 48-hour window - just in case of unforeseen problems.

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 28, 2006 1:03pm
Forum: feature_films Subject: Re: ground rules for uploading film

P.S. - FWIW, a PDF file is already in the upload directory of the new server - explaining exactly what I'm uploading.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffakb Date: Jul 28, 2006 1:17pm
Forum: feature_films Subject: Re: ground rules for uploading film

I remember someone posting a weird solution to the resume problem but I can't find it now. They were having the same problem you were having, some sort of error late in the upload process with autoresume in Smartftp.

They said that the partial upload was getting renamed to some weird name. It may begin with a "." which in Unix would make it appear hidden which may have been an additional complication. They suggested turning off autoresume and moving the weirdly named file back to the original name and manually resuming.

This is my flawed recollection, if someone can locate the post about this that would be great.

Oh, you shouldn't have to upload an mpeg1 file, those should be autogenerated by the Archives' intake system.

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 28, 2006 1:54pm
Forum: feature_films Subject: Re: ground rules for uploading film

You'll probably think I'm a member of the Flat Earth Society for suggesting this (grin) ... but another low-tech solution to this problem would be to allow people to burn files to a disc (DVD and/or CD) and (ahem) "mail" them to someone at the archive to copy onto the server. I imagine a "copy to" command would work faster than 45K/s, hehe.

In any case, just before I go to work tonight, I'm going to disconnect and reboot my system. If the resume function continues to function, I'll leave my computer running through the night - and should complete around 4:15 PM PDT. I'll immediately start the MPEG1 file upload which, I'm guessing, will take about 5 hours ... and follow that with the smaller WMV and RM files. I "should" be able to turn off my computer tomorrow night. And when I get home from work at about 9:00 AM, I'll still have two hours and 15 minutes to "check in" before the 48 hour window closes. Knock on wood for me.

Of course, if the resume doesn't work before I go to work tonight, I'll swear a little bit to myself, delete whatever's in the new server directory, and resolve to be a downloader-only in the future.

BTW, the reason I'm uploading the MPEG1 file is because (I know this from history), unless they've changed things here, the MPEG1 file they auto-generate is NOT a VCD-compliant MPEG1 file. Mine is (grin). Of course, that means I have one more thing to worry about ... that they might "auto-generate" my VCD compliant file into oblivion, replacing it with their own file (sigh). We'll see.

This post was modified by AlecWest on 2006-07-28 20:54:00

Reply to this post
Reply [edit]

Poster: k-otic Date: Jul 30, 2006 12:39am
Forum: feature_films Subject: Re: ground rules for uploading film

i guess that would be me .. but i didn`t suggest to rename the file as far as i can remember it :)
The easiest solution for this is just WAIT!
It can take maybe 10 to 20 minutes before the server writes the temp file with the weird name back to your account and you can resume your upload
In other words auto resume does not work so you better switch it off

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 1, 2006 9:44am
Forum: feature_films Subject: Re: ground rules for uploading film

Notice that in the very first post to this thread, I mentioned that shortly after 24 hours had passed by, I was disconnected by the server ... and attempts to resume did not work. While I can't say for sure, this may not be a server problem but a SmartFTP problem instead.

What happened was, when the 24 hour mark was reached, SmartFTP engaged "scheduling" - a block I had purposefully "unchecked." And when the FTP connected again, it treated the upload as a NEW upload as opposed to a RESUMED upload.

In any event, I'm making one more attempt to upload the MPEG2 file ... this time using CoreFTP-LE ... which does have the "resume" feature but does "not" have any fancy scheduling routines to muck things up.

I'll know the answer to this dilemma in 24 hours. If, in 24 hours, CoreFTP-LE continues to upload the file, I'm in business. On the other hand, if the upload stops at the 24 hour mark and cannot resume, then there is some sort of problem with archive servers being able to sustain a connection for more than 24 consecutive hours - meaning that attempts to upload a file that takes LONGER than that are fruitless and pointless. And if the latter is true, that would be most interesting - seeing as how DVD-ready files can be up to 4.3 gigs (like mine) and, at 42K/s, will ALWAYS take longer than 24 hours to complete.

Time will tell ...

Reply to this post
Reply [edit]

Poster: AlecWest Date: Jul 29, 2006 10:19am
Forum: feature_films Subject: Re: ground rules for uploading film

Editing my earlier grumpy post (grin). My last attempt to upload the MPEG2 file failed. At 3:14 AM, while I was at work, the server disconnected me ... and when SmartFTP reconnected, the "resume" feature didn't work (double-sigh). So, this is what I'll do now.

I'll "TRY TO" upload the 3 smaller files first - the VCD-compliant MPEG1 file, the WMV file for broadband streaming, and the RM file for modem streaming. Then, I'll "check in" afterward. But later, I'll try Diana's recommendation to "add" to the mix, quote:

"Successfully upload a file in a 48 hr period, and check it in. Then, go to edit details> item manager> checkout> upload another file as per instructions that time> checkin on that one. The repeat checkout> up > checking with next file. (6 files, 6 48 hr windows if you want.)"

This post was modified by AlecWest on 2006-07-29 17:19:24